Method, medium, and apparatus of filtering depth noise using depth information

ABSTRACT

A depth noise filtering method and apparatus is provided. The depth noise filtering method may perform spatial filtering or temporal filtering according to depth information. In order to perform spatial filtering, the depth noise filtering method may determine a characteristic of a spatial filter based on depth information. Also, in order to perform temporal filtering, the depth noise filtering method may determine a number of reference frames based on depth information. The depth noise filtering method may adaptively remove depth noise according to depth information and thereby enhance a noise filtering performance.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of Korean Patent Application No.10-2009-0005090, filed on Jan. 21, 2009, in the Korean IntellectualProperty Office, the disclosure of which is incorporated herein byreference.

BACKGROUND

1. Field

One or more embodiments relate to an image process to process athree-dimensional (3D) image, and more particularly, to a depth noisefiltering method and apparatus that may filter depth noise occurringaccording to a depth.

2. Description of the Related Art

As the demand for three-dimensional (3D) images increases, 3D camerascapable of collecting the 3D images have been appearing in the market.The 3D camera may obtain color information and depth informationassociated with an object to provide a 3D effect for a two-dimensional(2D) image. The depth information may be determined based on a distancebetween the object and the 3D camera.

Due to an image collection characteristic of the 3D camera, acharacteristic of depth noise included in the depth information may bedifferent depending on a distance from the object. When constructing a3D image based on depth information containing large depth noise, aquality of the 3D image may be deteriorated.

Accordingly, there is a need for a method that may remove depth noiseaccording to a distance between an object and a camera.

SUMMARY

According to an aspect of one or more embodiments, there may be provideda method of filtering depth noise, the method including obtaining depthinformation of a region that constitutes a filtering target frame of athree-dimensional (3D) image, generating a spatial filter for the regionusing the depth information, and performing depth noise filtering forthe 3D image by applying the spatial filter to the depth information ofthe region.

The depth information may correspond to a distance between an object andan image collection apparatus with respect to the region, and thegenerating may comprise determining at least one of a size of thespatial filter and a coefficient of the spatial filter to be applied tothe region, based on the depth information.

The generating may comprise determining the size of the spatial filterbased on the depth information, using at least one of the depthinformation, a measured maximum distance of the image collectionapparatus, a measured minimum distance of the image collectionapparatus, and a maximum size of the spatial filter in the measuredmaximum distance.

The generating may comprise generating the spatial filter correspondingto the depth information by referring to a table that stores spatialfilter information associated with at least one of the size of thespatial filter and the coefficient of the spatial filter that arepredetermined based on the depth information.

The generating may further comprise decreasing the size of the spatialfilter as the depth information corresponds to shorter distances.

The generating may further comprise increasing a decrease rate for thecoefficient of the spatial filter as the depth information correspondsto shorter distances, and the coefficient of the spatial filterdecreases while approaching a circumference of the spatial filter from acenter of the spatial filter.

The region may be any one of pixels or blocks that constitutes thefiltering target frame.

According to another aspect of one or more embodiments, there may beprovided a method of filtering depth noise, the method includingobtaining depth information of a region that constitutes a filteringtarget frame of a 3D image, calculating a number of reference framesassociated with the region using the depth information, and updating thedepth information of the region in the filtering target frame based onthe calculated number of reference frames.

A spatial filter may be applied to the depth information before thecalculating of a number of reference frames.

The updating may perform temporal filtering of the depth information.

The updating may further perform spatial filtering of the updated depthinformation.

The updating may further generate a spatial filter using the updateddepth information, and applies the spatial filter to the updated depthinformation.

The method may perform spatial filtering and then perform temporalfiltering on the depth information.

The method may perform temporal filtering and then perform spatialfiltering on the depth information.

The updating may further apply a weight to the depth information of theregion of each of the reference frames, the weight being based on thecalculated number of reference frames.

The depth information may correspond to a distance between an object andan image collection apparatus with respect to the region, and thecalculating may comprise calculating the number of reference framesbased on the depth information, using at least one of the depthinformation, a measured maximum distance of the image collectionapparatus, a measured minimum distance of the image collectionapparatus, and a number of reference frames in the measured maximumdistance.

The calculating may further comprise decreasing the number of referenceframes as the depth information of the region corresponds to shorterdistances.

The updating may further comprise averaging depth information of aregion of each of the reference frames to update the depth informationof the region in the filtering target frame.

According to still another aspect of one or more embodiments, there maybe provided a method of filtering depth noise, the method includingobtaining depth information of a region that constitutes a filteringtarget frame of a 3D image, generating a spatial filter for the regionusing the depth information to apply the spatial filter to the depthinformation of the region, calculating a number of reference framesassociated with the region using the depth information of the regionthat constitutes the filtering target frame, wherein the spatial filteris applied to the depth information, and updating the depth informationof the region in the filtering target frame based on the calculatednumber of reference frames.

The depth information may correspond to a distance between an object andan image collection apparatus with respect to the region, and thegenerating may comprise determining at least one of a size of thespatial filter and a coefficient of the spatial filter to be applied tothe region, based on the depth information.

The generating may further comprise generating the spatial filtercorresponding to the depth information by referring to a table thatstores spatial filter information associated with at least one of thesize of the spatial filter and the coefficient of the spatial filterthat are predetermined based on the depth information.

The generating may further comprise determining the size of the spatialfilter based on the depth information, using at least one of the depthinformation, a measured maximum distance of the image collectionapparatus, a measured minimum distance of the image collectionapparatus, and a maximum size of the spatial filter in the measuredmaximum distance.

The depth information may correspond to a distance between an object andan image collection apparatus with respect to the region, and thecalculating may comprise calculating the number of reference framesbased on the depth information, using at least one of the depthinformation, a measured maximum distance of the image collectionapparatus, a measured minimum distance of the image collectionapparatus, and a number of reference frames in the measured maximumdistance.

The updating may comprise averaging depth information of a region ofeach of the reference frames to update the depth information of theregion in the filtering target frame.

According to yet another aspect of one or more embodiments, there may beprovided a method of filtering depth noise, the method includingcalculating a number of reference frames with respect to a region thatconstitutes a filtering target frame of a 3D image, using depthinformation of the region, updating depth information of the region inthe filtering target frame based on the calculated number of referenceframes, and generating a spatial filter for the region using the updateddepth information to apply the spatial filter to the depth informationof the region.

The depth information may correspond to a distance between an object andan image collection apparatus with respect to the region, and thecalculating of the number of reference frames may comprise calculatingthe number of reference frames based on the depth information, using atleast one of the depth information, a measured maximum distance of theimage collection apparatus, a measured minimum distance of the imagecollection apparatus, and a number of reference frames in the measuredmaximum distance.

The updating may comprise averaging depth information of a region ofeach of the reference frames to update the depth information of theregion in the filtering target frame.

The depth information may correspond to a distance between an object andan image collection apparatus with respect to the region, and thegenerating may comprise determining at least one of a size of thespatial filter and a coefficient of the spatial filter to be applied tothe region, based on the updated depth information.

The generating may comprise generating the spatial filter correspondingto the updated depth information by referring to a table that storesspatial filter information associated with at least one of the size ofthe spatial filter and the coefficient of the spatial filter that arepredetermined based on the updated depth information.

The generating may comprise determining the size of the spatial filterbased on the updated depth information, using at least one of the depthinformation, a measured maximum distance of the image collectionapparatus, a measured minimum distance of the image collectionapparatus, and a maximum size of the spatial filter in the measuredmaximum distance.

According to a further another aspect of one or more embodiments, theremay be provided a method of generating a spatial filter, the methodincluding obtaining depth information of a region that constitutes afiltering target frame of a 3D image, determining a characteristic of aspatial filter for the region based on the depth information, andgenerating the spatial filter to be applied to the region based on thecharacteristic of the spatial filter.

The depth information may correspond to a distance between an object andan image collection apparatus with respect to the region, and thedetermining comprises determining at least one of a size of the spatialfilter and a coefficient of the spatial filter to be applied to theregion, based on the depth information.

The generating may comprise generating the spatial filter correspondingto the depth information by referring to a table that stores spatialfilter information associated with at least one of the size of thespatial filter and the coefficient of the spatial filter that arepredetermined based on the depth information.

The determining may comprise determining the size of the spatial filterbased on the depth information, using at least one of the depthinformation, a measured maximum distance of the image collectionapparatus, a measured minimum distance of the image collectionapparatus, and a maximum size of the spatial filter in the measuredmaximum distance.

The determining may comprise decreasing the size of the spatial filteras the depth information is updated to correspond to a shorter distance.

The determining may comprise increasing a decrease rate for thecoefficient of spatial filter as the depth information is updated tocorrespond to a shorter distance, and the coefficient of the spatialfilter decreases while approaching a circumference of the spatial filterfrom a center of the spatial filter.

According to still another aspect of one or more embodiments, there maybe provided an apparatus for filtering depth noise, the apparatusincluding a spatial filter generation unit to generate a spatial filterfor a region that constitutes a filtering target frame of a 3D image,using depth information of the region, and a filtering unit to performdepth noise filtering for the 3D image by applying the spatial filter tothe depth information of the region.

The depth information may correspond to a distance between an object andan image collection apparatus with respect to the region, and thespatial filter generation unit determines at least one of a size of thespatial filter and a coefficient of the spatial filter to be applied tothe region, based on the depth information.

The spatial filter generation unit may generate the spatial filtercorresponding to the depth information by referring to a table thatstores spatial filter information associated with at least one of thesize of the spatial filter and the coefficient of the spatial filterthat are predetermined based on the depth information.

According to still another aspect of one or more embodiments, there maybe provided an apparatus for filtering depth noise, the apparatusincluding a frame calculation unit to calculate a number of referenceframes associated with a region that constitutes a filtering targetframe of a 3D image, using first depth information of the region, and adepth information decision unit to determine second depth information ofthe region in the filtering target frame based on the calculated numberof reference frames.

The depth information decision unit may average depth information of aregion of each of the reference frames to determine the second depthinformation of the region in the filtering target frame.

The apparatus may perform temporal filtering using the second depthinformation.

The apparatus may perform spatial filtering using the second depthinformation.

The depth information decision unit may further determine third depthinformation by generating a spatial filter using the second depthinformation, and applying the spatial filter to the second depthinformation.

The apparatus may perform spatial filtering and then perform temporalfiltering.

The apparatus may perform temporal filtering and then perform spatialfiltering.

The depth information decision unit may apply a weight to the depthinformation of the region of each of the reference frames, and theweight may be based on the calculated number of reference frames.

The apparatus may apply a spatial filter to depth information togenerate the first depth information.

The apparatus may generate the spatial filter using depth informationother than the first and second depth information.

Additional aspects, features, and/or advantages of embodiments will beset forth in part in the description which follows and, in part, will beapparent from the description, or may be learned by practice of thedisclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

These and/or other aspects and advantages will become apparent and morereadily appreciated from the following description of the embodiments,taken in conjunction with the accompanying drawings of which:

FIG. 1 illustrates a process of reconstructing a three-dimensional (3D)image using a depth noise filtering apparatus according to anembodiment;

FIG. 2 is a flowchart illustrating a method of performing spatialfiltering to remove depth noise according to an embodiment;

FIG. 3 is a flowchart illustrating a method of performing temporalfiltering to remove depth noise according to an embodiment;

FIG. 4 is a flowchart illustrating a method of performing spatialfiltering and temporal filtering to remove depth noise according to anembodiment;

FIG. 5 is a flowchart illustrating a method of performing temporalfiltering and spatial filtering to remove depth noise according to anembodiment;

FIG. 6 illustrates an example of a depth noise difference based on depthinformation according to an embodiment;

FIG. 7 illustrates a process of calculating a size of a spatial filterbased on depth information according to an embodiment;

FIG. 8 illustrates a process of calculating a coefficient of a spatialfilter based on depth information according to an embodiment;

FIG. 9 illustrates an example of applying a spatial filter to depthinformation according to an embodiment;

FIG. 10 illustrates an example of determining a number of referenceframes based on depth information to thereby perform temporal filteringaccording to an embodiment;

FIG. 11 illustrates a configuration of a depth noise filtering apparatusto perform spatial filtering according to an embodiment; and

FIG. 12 illustrates a configuration of a depth noise filtering apparatusto perform temporal filtering according to an embodiment.

DETAILED DESCRIPTION

Reference will now be made in detail to the embodiments, examples ofwhich are illustrated in the accompanying drawings, wherein likereference numerals refer to the like elements throughout. Theembodiments are described below to explain the present disclosure byreferring to the figures.

FIG. 1 illustrates a process of reconstructing a three-dimensional (3D)image using a depth noise filtering apparatus 103 according to anembodiment. Also, the depth noise filtering apparatus 103 of FIG. 1 maybe applicable throughout this specification.

Referring to FIG. 1, an image collection apparatus 102 may photograph anobject 101 to collect images. Also, the image collection apparatus 102may collect 3D images from the object 101, that is, the image collectionapparatus 102 may obtain color information and depth informationassociated with the object 101. Here, the color information denotesinformation that is collected via a charge coupled device (CCD) or acomplementary metal-oxide semiconductor (CMOS) image sensor of the imagecollection apparatus 102. The depth information denotes a distance fromthe image collection apparatus 102 to each point of the object 101.

For example, the image collection apparatus 102 may measure a durationof time during which a light emitted from the image collection apparatus102 is reflected from the object 101 to come back to the imagecollection apparatus 102, and then measure the distance between theobject 101 and the image collection apparatus 102. The image collectionapparatus 102 may obtain the depth information through the measureddistance. In this instance, the obtained depth information may includedepth noise according to a distance between the object 101 and the imagecollection apparatus 102. The depth noise filtering apparatus 103 mayperform filtering to remove the depth noise in the depth information.Specifically, the depth noise filtering apparatus 103 may performfiltering using a low pass filter (LPF) to thereby remove the depthnoise.

For example, the depth noise filtering apparatus 103 may perform spatialfiltering according to the depth information. Also, the depth noisefiltering apparatus 103 may perform temporal filtering according to thedepth information. Also, the depth noise filtering apparatus 103 mayperform both spatial filtering and temporal filtering according to thedepth information. Also, the depth noise filtering apparatus 103 maysequentially perform spatial filtering and temporal filtering.

The depth noise filtering apparatus 103 may calculate a size or acoefficient of a filter based on the depth information to therebygenerate the filter, and may perform spatial filtering by applying thegenerated filter. The depth noise filtering apparatus 103 may calculatea number of reference frames based on the depth information to performtemporal filtering.

Specifically, an enhanced 3D image 104 may be generated byreconstructing the filtered depth information and color information.

As described above, the following methods of FIGS. 2 through 5 may beperformed by the depth noise filtering apparatus 103 of FIG. 1 and thusmay be described with reference to FIG. 1.

FIG. 2 is a flowchart illustrating a method of performing spatialfiltering to remove depth noise according to an embodiment.

Referring now to FIGS. 1 and 2, in operation S201, the depth noisefiltering apparatus 103 may obtain depth information of a region thatconstitutes a filtering target frame of a 3D image. Here, the region maybe pixels or blocks that constitute the filtering target frame.Specifically, the depth noise filtering apparatus 103 may performfiltering based on a pixel unit or a block unit. The depth informationmay denote a distance between the object 101 and the image collectionapparatus 102.

The depth noise filtering apparatus 103 may obtain, via the imagecollection apparatus 102, the depth information corresponding to thedistance between the object 101 and the image collection apparatus 102.

In operation S202, the depth noise filtering apparatus 103 may generatea spatial filter for the region using the depth information. Forexample, the depth noise filtering apparatus 103 may determine at leastone of a size of the spatial filter and a coefficient of the spatialfilter to be applied to the region, based on the depth information.

In this instance, the depth noise filtering apparatus 103 may determinethe size of the spatial filter based on the depth information, using thedepth information, a measured maximum distance of the image collectionapparatus 102, a measured minimum distance of the image collectionapparatus 102, and a maximum size of the spatial filter in the measuredmaximum distance. For example, the depth noise filtering apparatus 103may determine the size of the spatial filter according to the followingEquation 1.

$\begin{matrix}{{Size} = {\left( \frac{{depth} - {\min\mspace{14mu}{Distance}}}{\begin{matrix}{{\max\mspace{14mu}{Distance}} -} \\{\min\mspace{14mu}{Distance}}\end{matrix}} \right)*\max\mspace{14mu}{filtersize}}} & {{Equation}\mspace{14mu} 1}\end{matrix}$

Here, depth denotes the depth information, size denotes the size of thespatial filter according to the depth information, min Distance denotesthe measured minimum distance of the image collection apparatus 102, maxDistance denotes the measured maximum distance of the image collectionapparatus 102, and max filtersize denotes the maximum size of thespatial filter in the measured maximum distance of the image collectionapparatus 102.

For example, it is assumed here that the measured maximum distance ofthe image collection apparatus 102 is 3.5 m, the measured minimumdistance thereof is 0.5 m, and the maximum size of the spatial filter inthe measured maximum distance is 9. Also, it is assumed that an actualdepth image is between 1.5 m and 3 m. When the depth information is 1.5m, the size of the spatial filter may be determined as integer 3. Inthis case, the spatial filter may be determined as a 3×3 filter. Thecoefficient of the spatial filter may be determined as 1/9. Also, whenthe depth information is 1.9 m, the size of the spatial filter may bedetermined as 1.2, and the spatial filter may be determined as a 4×4filter. The coefficient of the spatial filter may be determined as 1/16.Here, the coefficient of the spatial filter may denote a coefficient ofa weight to be applied to the filtering target frame of the 3D image.

According to the above Equation 1, the depth noise filtering apparatus103 may decrease the size of the spatial filter as the depth informationcorresponds to shorter distances. Conversely, as the depth informationcorresponds to longer distances, the depth noise filtering apparatus 103may increase the size of the spatial filter.

For example, the depth noise filtering apparatus 103 may determine thecoefficient of the spatial filter based on the depth information. In thecase of the same filter size, the depth noise filtering apparatus 103may determine the coefficient of the spatial filter to be differentbased on the depth information.

Here, the coefficient of the spatial filter may decrease whileapproaching a circumference of the spatial filter from a center of thespatial filter. A decrease rate for the coefficient of the spatialfilter from the center of the spatial filter to the circumferencethereof may be differently determined. Specifically, as the depthinformation corresponds to shorter distances, the depth noise filteringapparatus 103 may determine the decrease rate for the coefficient of thespatial filter to be large. Conversely, as the depth informationcorresponds to longer distances, the depth noise filtering apparatus 103may determine the decrease rate for the coefficient of the spatialfilter to be small.

The decrease rate for the coefficient of the spatial filter may beexpressed by, for example, a Gaussian function. In this instance, awidth of the Gaussian function may be determined based on a sigma valueof the Gaussian function. As the depth information corresponds toshorter distances, the sigma value may be set to be small. Conversely,as the depth information corresponds to longer distances, the sigmavalue may be set to be large.

The depth noise filtering apparatus 103 may simultaneously determineboth the size and the coefficient of the spatial filter based on thedepth information. For example, the depth noise filtering apparatus 103may generate the spatial filter corresponding to the depth informationby referring to a table that stores spatial filter informationassociated with at least one of the size of the spatial filter and thecoefficient of the spatial filter that are predetermined based on thedepth information. Specifically, the depth noise filtering apparatus 103may pre-operate a characteristic of the spatial filter with respect toparticular depth information and then immediately generate the spatialfilter appropriate for input depth information.

When depth information D1 for generating the spatial filter is not setin the table, the depth noise filtering apparatus 103 may generate thespatial filter by combining predetermined depth information existing inthe table. Specifically, the depth noise filtering apparatus 103 maydetermine two pieces of depth information D2 and D3 capable of includingthe depth information D1, among the predetermined depth information ofthe table. The spatial filter of the depth information D1 may begenerated by applying a weight to a spatial filter of each of the depthinformation D2 and D3 based on a difference with the depth informationD1.

In operation S203, the depth noise filtering apparatus 103 may performdepth noise filtering for the 3D image by applying the spatial filter tothe depth information of the region. As described above, the region maybe pixels or blocks that constitute the filtering target frame. Depthnoise corresponding to a high frequency component may be removed byapplying the spatial filter. In particular, since the high frequencycomponent appears as depth noise as the depth information corresponds tolonger distances, the depth noise filtering apparatus 103 may remove thedepth noise by increasing a strength of the depth noise filtering. Anexample of applying the spatial filter will be described later in detailwith reference to FIG. 9.

Specifically, in a single filtering target frame that constitutes the 3Dimage, spatial filtering may indicate removing of depth noise byapplying the spatial filter to the region that constitutes the filteringtarget frame. Also, as a distance from the object 101 becomes farther,depth noise may more frequently occur. Accordingly, as the distance fromthe object 101 becomes farther, the depth noise filtering apparatus 103may increase the strength of filtering to thereby remove the depthnoise.

FIG. 3 is a flowchart illustrating a method of performing temporalfiltering to remove depth noise according to an embodiment.

Referring now to FIGS. 1 and 3, in operation S301, the depth noisefiltering apparatus 103 may obtain depth information of a region thatconstitutes a filtering target frame of a 3D image. In this instance,the depth noise filtering apparatus 103 may obtain the depth informationfrom the image collection apparatus 102. The region and the depthinformation are the same as the description made above with reference toFIG. 2.

In operation S302, the depth noise filtering apparatus 103 may calculatea number of reference frames associated with the region using the depthinformation. Since the depth information is different for each region,the number of reference frames calculated for each region may be alsodifferent. In this instance, a plurality of frames that appearstemporally previously or subsequently based on the filtering targetframe may be determined as the reference frames.

The depth noise filtering apparatus 103 may calculate the number ofreference frames based on the depth information, using at least one ofthe depth information, a measured maximum distance of the imagecollection apparatus 102, a measured minimum distance of the imagecollection apparatus 102, and a number of reference frames in themeasured maximum distance. For example, the depth noise filteringapparatus 103 may calculate the number of reference frames according tothe following Equation 2.

$\begin{matrix}{{Num} = {\left( \frac{{depth} - {\min\mspace{14mu}{Distance}}}{\begin{matrix}{{\max\mspace{14mu}{Distance}} -} \\{\min\mspace{14mu}{Distance}}\end{matrix}} \right)*\max\mspace{14mu}{Num}\mspace{14mu}{of}\mspace{14mu}{frames}}} & {{Equation}\mspace{14mu} 2}\end{matrix}$

Here, depth denotes the depth information, Num² equals the number ofreference frames, min Distance denotes the measured minimum distance ofthe image collection apparatus 102, max Distance denotes the measuredmaximum distance of the image collection apparatus 102, and max Num offrames denotes a maximum number of reference frames in the measuredmaximum distance of the image collection apparatus 102.

For example, it is assumed here that the measured maximum distance ofthe image collection apparatus 102 is 3.5 m, the measured minimumdistance thereof is 0.5 m, and the maximum number of reference frames inthe measured maximum distance is 25. Also, it is assumed that an actualdepth image is between 1.5 m and 3 m. When the depth information is 1.5m, Num may be determined as integer 3 and thus the number of referenceframes may become 9. Also, when the depth information is 1.9 m, Num maybe determined as 4.2 and thus the number of reference frames may become16.

In this instance, a weight to be applied to a region of each referenceframe may be determined to be in inverse proportion to the number ofreference frames. Specifically, when the number of reference frames is9, a weight 1/9 may be applicable to depth information of a region ofeach of nine reference frames.

According to the above Equation 2, the depth noise filtering apparatus103 may decrease the number of reference frames as the depth informationcorresponds to shorter distances and, conversely, may increase thenumber of reference frames as the depth information corresponds tolonger distances.

In operation S303, the depth noise filtering apparatus 103 may determinethe depth information of the region in the filtering target frame basedon the calculated number of reference frames. For example, the depthnoise filtering apparatus 103 may average depth information of thecalculated reference frames to determine depth information of the regionin the filtering target frame.

For example, when the number of reference frames is 9, the depth noisefiltering apparatus 103 may initially apply a weight 1/9 to depthinformation of a region of each of nine reference frames correspondingto the region of the filtering target frame. Next, the depth noisefiltering apparatus 103 may average the result of the application anddetermine the depth information of the region of the filtering targetframe.

FIG. 4 is a flowchart illustrating a method of performing spatialfiltering and temporal filtering to remove depth noise according to anembodiment.

Referring now to FIGS. 1 and 4, in operation S401, the depth noisefiltering apparatus 103 may obtain depth information of a region thatconstitutes a filtering target frame of a 3D image. The region and thedepth information may refer to the description made above with referenceto FIG. 2.

In operation S402, the depth noise filtering apparatus 103 may generatea spatial filter for the region using the depth information obtained inoperation S401 to apply the spatial filter to the depth information ofthe region. Here, operations S401 and S402 may indicate spatialfiltering. Further detailed description related to operations S401 andS402 may refer to the description made above with reference to FIG. 2.

In operation S403, the depth noise filtering apparatus 103 may calculatea number of reference frames associated with the region using the depthinformation of the region that constitutes the filtering target frame.Here, the spatial filter is applied to the depth information.

In operation S404, the depth noise filtering apparatus 103 may determineupdated depth information of the region in the filtering target framebased on the calculated number of reference frames which is calculatedusing the depth information obtained in S403. Here, operations S403 andS404 may indicate temporal filtering. Further detailed descriptionrelated to operations S403 and S404 may refer to the description madeabove with reference to FIG. 2.

Specifically, FIG. 4 may illustrate an example of initially applyingspatial filtering and subsequently applying temporal filtering withrespect to the 3D image.

FIG. 5 is a flowchart illustrating a method of performing temporalfiltering and spatial filtering to remove depth noise according to anembodiment.

Referring now to FIGS. 1 and 5, in operation S501, the depth noisefiltering apparatus 103 may obtain depth information of a region thatconstitutes a filtering target frame of a 3D image. The region and thedepth information may refer to the description made above with referenceto FIG. 2.

In operation S502, the depth noise filtering apparatus 103 may calculatea number of reference frames associated with the region that constitutesthe filtering target frame of the 3D image, using the depth informationof the region.

In operation S503, the depth noise filtering apparatus 103 may determinedepth information of the region in the filtering target frame based onthe calculated number of reference frames. Here, operations S502 andS503 may be associated with the aforementioned process of applyingtemporal filtering and thus further detailed description related theretomay refer to the description made above with reference to FIG. 3.

In operation S504, the depth noise filtering apparatus 103 may generatea spatial filter of the region using the depth information determined inoperation S503 and apply the spatial filter to the depth information ofthe region determined in operation S503.

Here, operation S504 may be associated with the aforementioned processof applying spatial filtering and thus further detailed descriptionrelated thereto may refer to the description made above with referenceto FIG. 3.

Specifically, FIG. 5 may illustrate an example of initially applyingtemporal filtering and subsequently applying spatial filtering withrespect to the 3D image.

FIG. 6 is a diagram illustrating an example of a depth noise differencebased on depth information according to an embodiment.

Referring to FIG. 6, it is possible to verify a level of depth noiseaccording to depth information that is a distance between an object 101and an image collection apparatus 102. As described above with referenceto FIG. 1, the depth information may be determined by dividing, by alight speed, a duration of time during which a light emitted from theimage collection apparatus 102 is reflected from the object 101 to comeback to the image collection apparatus 102. In this instance, when theobject 101 is positioned to be close to the image collection apparatus102, a light intensity of light reflected from the object 101 may belarge and thus an accuracy of the depth information may be high.Accordingly, as a distance between the object 101 and the imagecollection apparatus 102 becomes closer, depth noise may be relativelysmall. Conversely, when the object 101 is positioned to be away from theimage collection apparatus 102, the light intensity of the reflectedlight may be small and thus the accuracy of the depth information may below. In this instance, the depth noise against the depth information maybe relatively large.

Specifically, as the distance between the object 101 and the imagecollection apparatus 102 becomes farther, depth noise may increase.Conversely, as the distance between the object 101 and the imagecollection apparatus 102 becomes closer, depth noise may decrease.Accordingly, the depth noise filtering apparatus 103 of FIG. 1 accordingto an embodiment may generate the spatial filter so that the coefficientof the spatial filter may decrease while approaching a circumference ofthe spatial filter from a center of the spatial filter. In thisinstance, the depth noise filtering apparatus 103 may determine adecrease rate for the coefficient of the spatial filter to be differentbased on the depth information. For example, as the distance between theobject 101 and the image collection apparatus 102 becomes farther, thedepth noise filtering apparatus 103 may reduce the decrease rate for thecoefficient of the spatial filter. Conversely, as the distance betweenthe object 101 and the image collection apparatus 102 becomes closer,the depth noise filtering apparatus 103 may increase the decrease ratefor the coefficient of the spatial filter. Also, as the distance betweenthe object 101 and the image collection apparatus 102 becomes farther,the depth noise filtering apparatus 103 may increase the number ofreference frames for temporal filtering.

FIG. 7 illustrates a process of calculating a size of a spatial filterbased on depth information according to an embodiment.

As described above with reference to FIG. 6, the depth noise filteringapparatus 103 of FIG. 1 may increase the size of the spatial filter forspatial filtering as a distance between an object 101 and an imagecollection apparatus 102 becomes farther. Specifically, when thedistance between the object 101 and the image collection apparatus 102corresponds to longer distances, a 5×5 spatial filter 701 may bedetermined. When the distance between the object 101 and the imagecollection apparatus 102 corresponds to intermediate distances, a 4×4spatial filter 702 may be determined. Also, when the distance betweenthe object 101 and the image collection apparatus 102 corresponds toshort distances, a 3×3 spatial filter 703 may be determined.

The process of determining the size of the spatial filter may refer tothe above Equation 1.

FIG. 8 illustrates a process of calculating a coefficient of a spatialfilter based on depth information according to an embodiment.

It is assumed here that spatial filters 801, 802, 803, and 804 have thesame size.

As described above with reference to FIG. 6, the depth noise filteringapparatus 103 of FIG. 1 may reduce a decrease rate for the coefficientof the spatial filter as a distance between an object 101 and an imagecollection apparatus 102 becomes farther. Specifically, whileapproaching a circumference of the spatial filter from a center of thespatial filter, the depth noise filtering apparatus 103 may decrease thecoefficient of the spatial filter. In this instance, the depth noisefiltering apparatus 103 may differently determine the decrease rate forthe coefficient of the spatial filter based on the depth information.

As the depth information corresponds to short distances, the depth noisefiltering apparatus 103 may increase the decrease rate for thecoefficient of the spatial filter. When performing spatial filtering, aregion to be applied with the spatial filter may be less affected bydepth information of neighboring regions. Conversely, as the depthinformation corresponds to longer distances, the depth noise filteringapparatus 103 may decrease the decrease rate for the coefficient of thespatial filter. Specifically, when performing spatial filtering, theregion to be applied with the spatial filter may be greatly affected bythe depth information of the neighboring regions.

Referring to FIG. 8, the decrease rate for the coefficient of thespatial filter may be expressed by a Gaussian function. Here, a sigmavalue to determine a width of the Gaussian function denotes the decreaserate for the coefficient of the spatial filter, and may be set to bedifferent based on the depth information. For example, when the depthinformation corresponds to short distances, the sigma value of theGaussian function may be set to be small. Conversely, when the depthinformation corresponds to long distances, the sigma value of theGaussian function may be set to be large.

Referring to FIG. 8, as a distance between an object 101 and an imagecollection apparatus 102 becomes farther, depth noise may be strong andthus the depth noise filtering apparatus 103 may increase a filteringstrength. Specifically, as the filtering strength increases, it ispossible to remove a large amount of high frequency components in thedepth information. Conversely, as the distance between the object 101and the image collection apparatus 103 becomes closer, depth noise maybe weak and thus the depth noise filtering apparatus 103 may decreasethe filtering strength. For example, when the depth informationcorresponds to long distances, a high frequency component of a 3D imagemay be a noise component. In this instance, the depth noise filteringapparatus 103 may adjust the filtering strength to thereby remove thehigh frequency component corresponding to the noise component.Conversely, when the depth information corresponds to short distances,the high frequency component of the 3D image may be an originalcomponent of the 3D image, not the noise component. Accordingly, thedepth noise filtering apparatus 103 may adjust the filtering strength tothereby remain the high frequency component corresponding to theoriginal component of the 3D image.

FIG. 9 illustrates an example of applying a spatial filter 901 to depthinformation according to an embodiment.

The spatial filter 901 denotes a spatial filter that is generated by thedepth noise filtering apparatus 103 of FIG. 1. The depth information 902denotes depth information of a region to be applied with the spatialfilter 901. In this instance, the spatial filter 901 may be applied to aregion e. The region e may be pixels or a block that is composed of thepixels.

Another depth information 903 may be obtained by applying the spatialfilter 901 to the depth information 902 of the region. Specifically, thedepth information 903 corresponds to a result of performing spatialfiltering. e′ denotes depth information where spatial filtering isperformed.

For example, the depth information e′ where spatial filtering isperformed may be determined according to the following Equation 3e′=(A×a)+(B×b)+(C×c)+(D×d)+(E×e)+(F×f)+(G×g)+(H×h)+(I×i).  Equation 3:

Referring to the above Equation 3, spatial filtering may be performedbased on depth information of neighboring regions of the region to beapplied with the spatial filter 901. The above filtering process may bereferred to as a mask-based filtering process. According to anembodiment, a size or a coefficient of a spatial filter may bedetermined based on depth information of a filtering target region.

FIG. 10 illustrates an example of determining a number of referenceframes based on depth information to perform temporal filteringaccording to an embodiment.

A plurality of frames of FIG. 10 may denote the reference frames toperform temporal filtering. Here, it is assumed that a frame 1001 is afiltering target frame. The filtering target frame 1001 may includeregions 1002, 1003, and 1004.

Also, it is assumed here that depth information of the region 1002 isobtained from the object 101 of FIG. 1 that is positioned to be awayfrom the image collection apparatus 102, depth information of the region1003 is obtained from the object 101 (FIG. 1) that is positioned to bein an intermediate distance from the image collection apparatus 102(FIG. 1), and depth information of the region 1004 is obtained from theobject 101 (FIG. 1) that is positioned to be in a close distance fromthe image collection apparatus 102 (FIG. 2).

According to an embodiment, the depth noise filtering apparatus 103(FIG. 1) may calculate a number of reference frames associated with theregion using the depth information. For example, the depth noisefiltering apparatus 103 (FIG. 1) may increase the number of referenceframes as depth information of the region corresponds to longerdistances, and conversely, may decrease the number of reference framesas the depth information of the region corresponds to shorter distances.

Referring to FIG. 10, in the case of the region 1004, the depthinformation may correspond to short distances and thus a relativelysmall number of reference frames 1007 may be determined. Also, in thecase of the region 1003, the depth information may correspond tointermediate distances and thus an intermediate number of referenceframes 1006, greater than the number of reference frames 1007, may bedetermined. Also, in the case of the region 1002, the depth informationmay correspond to long distances and thus a relatively large number ofreference frames 1005 may be determined. In this instance, it can beseen that the number of reference frames 1005 is greater than the numberof reference frames 1006 or 1007.

Specifically, according to an embodiment, in the filtering target frame1001, the depth noise filtering apparatus 103 may decrease a number ofreference frames as depth information corresponds to shorter distancesand may increase the number of reference frames as the depth informationcorresponds to longer distances. For example, when performing temporalfiltering for the region 1003, the depth noise filtering apparatus 103may update the depth information of the region 1003 by averaging depthinformation of the region 1003 in a reference frame corresponding to thenumber of reference frames 1006.

FIG. 11 is a block diagram illustrating a configuration of a depth noisefiltering apparatus 1103 to perform spatial filtering according to anembodiment.

Referring to FIG. 11, the depth noise filtering apparatus 1103 mayinclude a spatial filter generation unit 1101 and a filtering unit 1102.

The spatial filter generation unit 1101 may generate a spatial filterfor a region that constitutes a filtering target frame of a 3D image,using depth information of the region. For example, the spatial filtergeneration unit 1101 may calculate at least one of a size and acoefficient of the spatial filter, using the depth information. Also,the spatial filter generation unit 1101 may simultaneously calculateboth the size and the coefficient of the spatial filter. As describedabove with reference to FIG. 1, the region may be pixels or blocks thatconstitute the filtering target frame. Depth information may correspondto a distance between an image collection apparatus and an object.

In this instance, the spatial filter generation unit 1101 may determinethe size of the spatial filter based on the depth information, using thedepth information, a measured maximum distance of the image collectionapparatus, a measured minimum distance of the image collectionapparatus, and a maximum size of the spatial filter in the measuredmaximum distance.

Also, the spatial filter generation unit 1101 may decrease the size ofthe spatial filter as the depth information corresponds to shorterdistances, and conversely, may increase the size of the spatial filteras the depth information corresponds to longer distances. Also, thespatial filter generation unit 1101 may generate the spatial filter sothat the coefficient of the spatial filter may decrease whileapproaching a circumference of the spatial filter to a center of thespatial filter. For example, the spatial filter generation unit 1101 maydetermine the decrease rate to be different according to the depthinformation. Specifically, the spatial filter generation unit 1101 mayincrease the decrease rate for the coefficient of the spatial filter asthe depth information corresponds to shorter distances. Conversely, asthe depth information corresponds to longer distances, the spatialfilter generation unit 1101 may decrease the decrease rate for thecoefficient of the spatial filter.

Specifically, the spatial filter generation unit 1101 may generate thespatial filter in real time by performing an operation process based onthe depth information. Also, for example, the spatial filter generationunit 1101 may generate the spatial filter corresponding to the depthinformation by referring to a table that stores spatial filterinformation associated with at least one of the size of the spatialfilter and the coefficient of the spatial filter that are predeterminedbased on the depth information. Specifically, the spatial filtergeneration unit 1101 may pre-operate a characteristic of the spatialfilter with respect to particular depth information and then immediatelygenerate the spatial filter appropriate for input depth information.

In this instance, when the table does not store depth information D1 forgenerating the spatial filter, the spatial filter generation unit 1101may generate the spatial filter by combining predetermined depthinformation existing in the table. Specifically, the spatial filtergeneration unit 1101 may generate a spatial filter according to thedepth information D1 by applying a weight to a spatial filtercorresponding to each of two pieces of depth information D2 and D3capable of including the depth information D1.

The filtering unit 1102 may perform depth noise filtering for the 3Dimage by applying the spatial filter to the depth information of theregion. In this instance, depth noise filtering may be performed basedon depth information of neighboring regions of the region by applying amask-based spatial filter.

FIG. 12 is a block diagram illustrating a configuration of a depth noisefiltering apparatus 1203 to perform temporal filtering according to anembodiment.

Referring to FIG. 12, the depth noise filtering apparatus 1203 mayinclude a frame calculation unit 1201 and a depth noise decision unit1202.

The frame calculation unit 1201 may calculate a number of referenceframes associated with a region that constitutes a filtering targetframe of a 3D image, using the depth information of the region. Forexample, the frame calculation unit 1201 may calculate the number ofreference frames based on the depth information, using the depthinformation, a measured maximum distance of the image collectionapparatus, a measured minimum distance of the image collectionapparatus, and a number of reference frames in the measured maximumdistance. In this instance, as the depth information of the regioncorresponds to longer distances, the frame calculation unit 1201 mayincrease the number of reference frames. Conversely, as the depthinformation of the region corresponds to shorter distances, the framecalculation unit 1201 may decrease the number of reference frames.

The depth information decision unit 1202 may determine the depthinformation of the region in the filtering target frame based on thecalculated number of reference frames. For example, the depthinformation decision unit 1202 may average depth information of a regionof each of the reference frames to determine the depth information ofthe region in the filtering target frame.

Matters not described in FIGS. 11 and 12 may refer to descriptions madeabove with reference to FIGS. 1 through 10.

According to an embodiment, there may be provided a depth noisefiltering method and apparatus that may perform spatial filteringaccording to depth information and thereby adaptively remove depthnoise.

According to an embodiment, there may be provided a depth noisefiltering method and apparatus that may perform temporal filteringaccording to depth information and thereby adaptively remove depthnoise.

According to an embodiment, there may be provided a depth noisefiltering method and apparatus that may calculate a size or acoefficient of a spatial filter according to depth information in orderto perform spatial filtering and thereby effectively remove depth noiseaccording to the depth information.

According to an embodiment, there may be provided a depth noisefiltering method and apparatus that may determine a number of referenceframes based on depth information in order to perform temporal filteringand thereby effectively remove depth noise according to the depthinformation.

The depth noise filtering method according to the above-describedembodiments can also be implemented through computer readablecode/instructions in/on a medium, e.g., a computer readable medium, tocontrol at least one processing element to implement any above describedembodiment. The medium can correspond to any medium/media permitting thestoring and/or transmission of the computer readable code.

The computer readable code can be recorded/transferred on a medium in avariety of ways, with examples of the medium including recording media,such as magnetic storage media (e.g., ROM, floppy disks, hard disks,etc.) and optical recording media (e.g., CD-ROMs, or DVDs), andtransmission media such as media carrying or including carrier waves, aswell as elements of the Internet, for example. Thus, the medium may besuch a defined and measurable structure including or carrying a signalor information, such as a device carrying a bitstream, for example,according to embodiments of the present invention. The media may also bea distributed network, so that the computer readable code isstored/transferred and executed in a distributed fashion. Still further,as only an example, the processing element could include a processor ora computer processor, and processing elements may be distributed and/orincluded in a single device.

Although a few embodiments have been shown and described, it would beappreciated by those skilled in the art that changes may be made inthese embodiments without departing from the principles and spirit ofthe disclosure, the scope of which is defined by the claims and theirequivalents.

What is claimed is:
 1. A method of filtering depth noise, the methodcomprising: obtaining depth information of a region that constitutes afiltering target frame of a three-dimensional (3D) image; generatingautomatically a spatial filter for the region using the depthinformation; and performing depth noise filtering for the 3D image byapplying the generated spatial filter to the depth information of theregion, wherein the generating comprises determining at least one of asize of a spatial filter and a coefficient of the spatial filter to beapplied to the region, based on the depth information, wherein the sizeof the spatial filter for a particular pixel is related to a depth valuefor the particular pixel in depth information, and wherein strength ofthe coefficient of the spatial filter for a particular pixel is relatedto the depth value for the particular pixel in depth information.
 2. Themethod of claim 1, wherein: the depth information corresponds to adistance between an object and an image collection apparatus withrespect to the region.
 3. The method of claim 2, wherein the generatingcomprises determining the size of the spatial filter based on the depthinformation, using at least one of the depth information, a measuredmaximum distance of the image collection apparatus, a measured minimumdistance of the image collection apparatus, and a maximum size of thespatial filter in the measured maximum distance.
 4. The method of claim2, wherein the generating comprises generating the spatial filtercorresponding to the depth information by referring to a table thatstores spatial filter information associated with at least one of thesize of the spatial filter and the coefficient of the spatial filterthat are predetermined based on the depth information.
 5. The method ofclaim 2, wherein the generating further comprises decreasing the size ofthe spatial filter as the depth information corresponds to shorterdistances.
 6. The method of claim 1, wherein the region is any one ofpixels or blocks that constitute the filtering target frame.
 7. Anon-transitory computer-readable recording medium storing a program forcausing at least one processor to implement the method of claim
 1. 8. Amethod of filtering depth noise, the method comprising: obtaining depthinformation of a region that constitutes a filtering target frame of athree-dimensional (3D) image; generating a spatial filter for the regionusing the depth information; and performing depth noise filtering forthe 3D image by applying the spatial filter to the depth information ofthe region, wherein: the depth information corresponds to a distancebetween an object and an image collection apparatus with respect to theregion, and the generating comprises determining at least one of a sizeof the spatial filter and a coefficient of the spatial filter to beapplied to the region, based on the depth information, and wherein: thegenerating further comprises increasing a decrease rate for thecoefficient of the spatial filter as the depth information correspondsto shorter distances, and the coefficient of the spatial filterdecreases while approaching a circumference of the spatial filter from acenter of the spatial filter.
 9. An apparatus for filtering depth noise,the apparatus comprising: a spatial filter generation unit toautomatically generate a spatial filter for a region that constitutes afiltering target frame of a 3D image, using depth information of theregion; and a filtering unit to perform depth noise filtering for the 3Dimage by applying the generated spatial filter to the depth informationof the region, wherein the spatial filter generation unit determines atleast one of a size of a spatial filter and a coefficient of a spatialfilter to be applied to the region, based on the depth information,wherein the size of the spatial filter for a particular pixel is relatedto a depth value for the particular pixel in depth information, andwherein strength of the coefficient of the spatial filter for aparticular pixel is related to the depth value for the particular pixelin depth information.
 10. The apparatus of claim 9, wherein: the depthinformation corresponds to a distance between an object and an imagecollection apparatus with respect to the region.
 11. The apparatus ofclaim 10, wherein the spatial filter generation unit generates thespatial filter corresponding to the depth information by referring to atable that stores spatial filter information associated with at leastone of the size of the spatial filter and the coefficient of the spatialfilter that are predetermined based on the depth information.